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ESTIMATION OF MOVEMENTS IN VIDEO IMAGES 

BACKGROUND OF THE INVENTION 
Field Of The Invention 

[0001] The invention relates to a method of motion estimation in 
video image data, in which, starting from a first and a second 
5 video image, parameter sets of two or more motion models are 

initially determined, and image objects are assigned to the motion 
models. The invention also relates to a device for performing the 
method, a device operating in accordance with the method for 
displaying video images, and a computer program product for motion 

10 estimation. 

[0002] Advances in multimedia techniques have led to the 
development of a multitude of video formats and display standards. 
They are distinguished, inter alia, also by their image rate, i.e., 
the number of frames per unit of time. When a video sequence is to 

15 be displayed on a PC or TV display screen, it is necessary to adapt 
to the image rate of the display apparatus. Interfaces suitable for 
this purpose operate by conversion methods of different complexity. 
The simplest method is to repeat (or omit) frames of the video 
sequence in the display, dependent on the desired image rate. 

20 However, when displaying video data thus treated, unwanted 

artifacts occur. Unwanted display errors occur dependent on the 
ratio of the image rates involved. The display appears to be 
jittery and irregular so that the motions displayed in the video 
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sequence have an unnatural effect. More elaborate methods perform 
an interpolation between consecutive video images, in which an 
algorithm for motion estimation is used, this algorithm initially 
recognizing the displacements of individual pixels from one image 
5 to the other, and generating image data therefrom which are 

temporally present between the images of the video sequence. The 
use of such methods in apparatus for home use requires the 
fundamental algorithms to supply a qualitatively high-value image 
rate conversion and require only a small number of computations 
10 because the digital signal processing electronics in apparatus for 
home use have a limited efficiency. 

[0003] Motion estimation methods of the type described in the 
opening paragraph are not only suitable for image rate conversion, 
but also for coding and compression in the transmission of video 
15 data, as well as for depth estimation in 3D image data processing, 
and for disparity estimations in stereo images. 

Description Of The Related Art 

[0004] Such a method is proposed in International Patent 
20 Application No. WO 99/16251, corresponding to U.S. Patent 

6,385,245. It is an efficient, object-oriented method of motion 
estimation in which two or more motion models are used so as to 
describe the displacement of image objects between a current and a 
previous video image. The motion models are determined by parameter 
2 5 sets from which displacement vectors can be computed. One of the 
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motion models is needed to deal with those image parts which are 
static. The associated displacement vector is thus the zero vector. 
The parameter sets of the other motion models are determined by 
evaluating the match errors of the motion models in the description 
5 of the displacement of image objects between consecutive video 

images. For the interpolation, it is then necessary to segment the 
image data and assign appropriate motion models to the individual 
image objects. The result of the segmentation is separate objects, 
i.e., image parts which perform a similar or comparable 

10 displacement from the previous to the current video image. 

[0005] The known motion estimation method is an efficient 
alternative to the otherwise conventional block-oriented method 
because the number of independently movable objects is small in 
normal video sequences and, consequently, only a correspondingly 

15 small number of motion models need be processed. A small number of 
computations results therefrom, which renders the method 
universally usable, also for home use. 

SUMMARY OF THE INVENTION 
20 [0006] The fundamental object of the present invention is to 

further improve the known motion estimation method and, 

simultaneously, further reduce the complexity. 

[0007] An important step in the motion estimation is the 

determination of the parameter sets for the motion models. In the 
25 known method, the parameter sets are combined to vectors. For each 
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motion model, a parameter set is selected from a quantity of 
candidate vectors in accordance with a selection criterion. The 
selection criterion consists of the evaluation of a match error. 
This is computed as the sum of absolute differences of individual 
5 motion-compensated pixel intensities between the current and the 
previous video image, while a displacement vector in accordance 
with one of the candidate vectors is used for compensation. An 
essential problem is that it is not clear in advance which motion 
model is to be assigned to which image area and with which 

10 parameter set. The known method is performed in such a way that the 
above-mentioned selection criterion is initially used with all 
motion models for the same image areas. Then, without an assignment 
being fixed, the best fitting parameter sets are selected. 
[0008] In accordance with the above-mentioned envisaged object, 

15 a further reduction of complexity in a motion estimation method of 
the type described in the opening paragraph, is achieved in that 
only parts of the image area are taken into account when 
determining the parameter sets. A problem then is that 
corresponding parts of the image area are to be selected in an 

2 0 appropriate manner so that displacements between the video images 
are as completely captured as possible. According to the invention, 
only those parts of the image area are thus taken into account for 
determining the parameter sets, in which the first video image is 
significantly distinguished from the second video image. 
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[0009] Such distinctions are a clear indication that there is a 
displacement at the corresponding locations. Image parts for 
determining the parameter sets can thereby be selected very easily 
without initially having to know more precise motion data. 
5 Moreover, it is avoided that stationary image parts are processed 
when determining the parameter sets for the motion models, for 
which stationary parts motion compensation is useless and, 
therefore, unnecessary. In fact, the parameters must be determined 
only in the non-stationary image parts. 

10 [0010] Since the selection criterion is only used for parts of 
the image area, the number of required computations is greatly 
reduced so that the overall motion estimation is accelerated. Based 
on the fact that only some hardly moving objects are displayed in 
typical video sequences, it is sufficient, under normal 

15 circumstances, to limit oneself to a corresponding number of 
"interesting" points in the video image when determining the 
parameter sets. 

[0011] The "interesting" image areas are suitably determined in 
that deviations between the video images are evaluated block by 

2 0 block, taking those blocks for determining the parameter sets into 
account in which the value of the deviation exceeds a predetermined 
threshold value. The image area is thus divided into individual 
blocks whose size should be dimensioned in such a way that the 
parameter sets can be determined by means of individual blocks. The 

2 5 deviations between the current and the previous video image may be 
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determined, for example, by forming the absolute differences of the 
pixel intensities, each time, within the individual blocks. The 
result is a positive number so that it can be easily ascertained, 
by comparison with a predetermined threshold value, whether there 
5 is motion or no motion in the associated image area. When 
determining the parameter sets, the method according to the 
invention is limited to those blocks in which a given distinction 
between the two video images can be recognized on the basis of the 
pixel intensities . 

10 [0012] This method has the additional advantage that the 

threshold value can be determined on the basis that the number of 
image areas taken into account for determining the parameter sets 
is limited to a predeterminable value. Since the overall method is 
to be performed in real time for the motion estimation, it should 

15 be ensured that the number of computations remains below a fixed 

maximum value. It is thus possible to adjust the threshold value in 
the case of repeated use of the method according to the invention 
in such a way that the data processing time remains uncritical. 
[0013] Practice has proven that it may be advantageous to take 

2 0 into account those parts of the image area for determining the 

parameter sets, in which motion was determined in previous video 
image data of a sequence of video images . A higher temporal 
consistency in the motion compensation is obtained in this way. 
[0014] For performing the method according to the invention, a 

25 device for motion estimation in video image data is suitable, this 
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device comprising a digital image memory in which a first and a 
second video image can be stored, and means for determining 
parameter sets of two or more motion models in accordance with a 
selection criterion. The device according to the invention 
5 comprises means for block-wise evaluation of the deviations between 
the current and the previous video image, and for selection of 
those blocks for use of the selection criterion, in which the value 
of the deviation exceeds a predeterminable threshold value. Such 
devices may be used, for example, as components in television and 

10 video apparatuses. The digital image memory of the device according 
to the invention, need not necessarily have a sufficient capacity 
for recording the first and the second video images simultaneously. 
The consecutive storage of the respective images is sufficient for 
the method according to the invention. 

15 [0015] Devices for displaying video images such as, for example, 
televisions, monitors, etc., comprising a digital image memory in 
which video image data can be stored, and electronic means for 
processing the image data stored in the image memory and for 
displaying video images on a display device, the means for 

2 0 processing the image data comprising means for determining 

parameter sets of two or more motion models in accordance with a 
selection criterion, may advantageously benefit from the method 
according to the invention when the means for processing the image 
data further comprise means for block-wise evaluation of the 

2 5 deviations between the current and the previous video image, and 
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for selection of those blocks for use of the selection criterion, 
in which the value of the deviation exceeds a predeterminable 
threshold value. Conventional, digitally operating televisions and 
monitors may be operated in a simple manner in accordance with the 
5 method according to the invention, with an improvement of the 
quality of the image displayed. Devices in the sense mentioned 
above are, for example, the cathode ray tubes or dot matrix 
displays conventionally used in televisions and monitors. Other 
devices for visual display of digital image data are also feasible. 

10 [0016] According to the invention, a computer program product is 
suitable for interpolation between pairs of video image data sets, 
this product comprising, as input, a first and a second video image 
and, starting therefrom, computes parameter sets of two or more 
motion models and supplies motion data describing the displacement 

15 of image objects from the previous to the current image, while the 
image data of the two video images are compared with each other and 
only those parts of the image area in which there are significant 
differences between the two video images are taken into account in 
the computation of the parameter sets. The computer program product 

2 0 may be made available on various data carriersm such asm diskettes, 
CD-ROMs or the like, but also for transfer via computer networks 
(for example, Internet) . 
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BRIEF DESCRIPTION OF THE DRAWINGS 
[0017] These and other aspects of the invention are apparent 
from and will be elucidated with reference to the embodiments 
described hereinafter. 
5 [0018] In the drawings: 

[0019] Fig. 1 shows a selection of interesting image areas; 
[0020] Fig. 2 is a block diagram of a motion estimation method 
according to the invention; and 

[0021] Fig. 3 is a block diagram of a device according to the 
10 invention, for displaying video images. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0022] When determining the parameter sets for the motion models 
according to the invention, a selection criterion is applied to the 
15 selected image areas. The selection criterion consists of, for 

example, the evaluation of a match error e. This is computed as the 
sum of absolute differences of individual motion-compensated pixel 
intensities between a current and a previous video image in the 
following manner: 



20 



*(C 0 , n) = £ W 0 (x} | F s (5, n) -F s (x- C 0 (x, n), n-l)\ 

xel(n) 



[0023] A summation is subsequently effected via image co- 



ordinates x - 



, comprised in a quantity I (n) of selected image 
25 areas. The absolute differences between the pixel intensities in 
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the current video image and the previous video image are added to 
these image co-ordinates. F s (x,n) is the pixel intensity at the 
image co-ordinate x in a video image with a reduced raster. It has 
proved that in determining the parameter, the use of a resolution- 
5 reduced (sub-sampled) image is entirely sufficient. This 

advantageously leads to a considerable reduction of the number of 
computations .. Due to the continuing index n, the number of the 
frame, and hence, the instant within the video sequence, is 
indicated. C 0 (x,n) indicates, for the image pair n, as a current 

10 video image and n-1 as a previous video image, the displacement 
vector at the image co-ordinate x in accordance with the motion 
model with the index o. W 0 {x) represents a weighting factor with 
which it is taken into account which motion model o was assigned to 
the image co-ordinates x in earlier image data of the video 

15 sequence. A combination between the determination of the parameters 
and image segmentation can thereby be realized, which has 
advantages with a view to the temporal consistency of the motion 
estimation and the efficiency of the method. 

[0024] Starting from four parameters, displacement vectors can 
20 be computed by means of the following motion model: 

, (s x (o,n) + X'd x (o,n)^ 
C Ax,n) = 

\s y (o 9 ri) + y'd y (o 9 ri)j 
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[0025] 



This is a simple linear first-order model with which 



translations and scalings can be described. The model is determined 
by the parameter set : 



The parameter set is determined in such a way that the above- 
mentioned match error for the corresponding motion model o assumes 
a minimal value. In the motion estimation method according to the 
10 invention, at least two motion models are used every time, one of 
which always has the zero vector as a parameter set so that the 
stationary image areas are described by this motion model with the 
displacement vector C 0 (x,n) = 0 . 

[0026] The next step in the motion estimation according to the 
15 invention is the image segmentation, i.e., assigning image areas to 
the motion models. To this end, the overall image area is initially 
subdivided into blocks. In practice, quadratic blocks of 8x8 pixels 
have proven to be suitable. For all image co-ordinates within the 
blocks at the position X , it then holds that xeB(X). For each 
2 0 block, a match error is again computed on a motion model o: 



25 determined by a. In the simplest case, that motion model o is 




s 0 (X, n)= £ | F s (x + (1 - a)C 0 (5, n\ n) -F s (x- aC 0 (x,n) 9 n - 1) | 



xeB(X) 



[0027] 



The instant when the segmentation should be valid is 
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assigned to the block X for which e o (X,n) is minimal. The 
assignment is then filed in the segmentation mask M(X,n) . 
[0028] According to the invention, the motion estimation method 
for determining the parameters of the motion models is limited to 
5 "interesting" image areas, given by the quantity I (n) . It is 

advantageous to fill the quantity I (n) with those blocks that are 
in poor conformity with the corresponding blocks in a previous 
image. This may be effected, for example, in accordance with the 
following prescription: 

10 

I(n) = {x\e 0 (X,n-\)>T} 

T is a predeterminable threshold value which fixes the extent of 
the deviation between two consecutive images as from which the 

15 parameters are determined in the relevant image area. 

[002 9] Fig. 1 is a video image 1 showing a motorcyclist 2 riding 
on a street 3. The motorcyclist moves from left to right in the 
section of the image. The background, thus also the street 3, is 
stationary. In the Figure, the selection of "interesting" image 

20 areas can be recognized and are shown as white blocks 4. A motion 
model describing the motion of the motorcyclist 2 is assigned to 
the white blocks 4. The image background is stationary and is 
assigned to another corresponding motion model. 

[0030] Fig. 2 shows, diagrammatically , the motion estimation 
25 procedure in accordance with the invention. Starting from a 



ST14GOA0.GOR 



PHNL 000643 

previous video image 6, a current video image 7 and a threshold 
value 8, image areas that are interesting for determining the 
parameter sets are selected in a first step 9 of the method 
described above. All of these image areas are provided with 
5 weighting factors 10 for a plurality of motion models and 
subsequently further processed in a step 11, in which the 
parameters of the motion models are determined in accordance with a 
selection criterion. Starting from the completely determined motion 
models, the overall image area is then subdivided into blocks in a 

10 step 12, and the displacement vectors corresponding to the 

individual motion models are computed for each block. Subsequently, 
the image area is segmented in step 13, in which the blocks are 
assigned to the motion models. The assignments, which are included 
in the weighting 10 for the next image pair, are stored in a 

15 segmentation mask 14 which is then obtained. 

[0031] Fig. 3 shows, diagrammatically , the structure of a 
digitally operating device which may be, for example, a television 
or a video monitor. The device receives a video signal 2 0 which is 
stored and prepared in a digital image processing unit 21. To this 

20 end, the image processing unit comprises an image memory 22, a 

processor 23 and a program memory 24. These elements may also be at 
least partly combined in a discrete component. The processor 23 
runs through a program stored in the program memory 24, this 
program controlling the image processing method according to the 

25 invention. A display unit 25 receives image data 26 prepared by the 
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image processing unit 21 and generates a signal 27 therefrom for 
driving a cathode ray tube 28 via which the video images are 
visually presented. 
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